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ABSTRACT — The main reason for the decrease in changes in appearance of the user is due to the factors 
such as ageing, beard growth, sun tan. Heterogeneous face recognition involves matching two face images from 
alternate imaging aesthetic forms, such as an infrared image to a photograph or a sketch to a photograph. A 
generic HFR framework is proposed in which both probe and gallery images are represented in terms of 
nonlinear similarities to a collection of prototype face images. The prototype subjects have an image in each 
modality (probe and gallery), and the similarity between an image and prototype images are measured. The 
features of this nonlinear prototype are projected into a linear discriminant subspace which increases the 
accuracy of this nonlinear prototype representation. In HFR framework we introduce Random sampling to 
control the small sample size problem which arises as a challenge. The excellence of the proposed approach is 
demonstrated in the experiment result as prototype random subspace. Previous studies have shown that the 
accuracy of Face Recognition Systems (FRSs) decreases with the time elapsed between enrollments and testing. 
So we have proposed Gabor filter along with SVM for Feature Extraction and Robustness. 

KEY WORDS — nonlinear similarity, local descriptors, ambiences, Random subspaces, Mugshot, infrared 
image, discriminant analysis, sketches, SVM, Gabor Filter. 



I. INTRODUCTION 

Face Recognition in recent days involves matching between heterogeneous image ambiences. When 
different scenarios can provide plausible solutions for difficult scenarios then it is coined as Heterogeneous face 
recognition. Heterogeneous face recognition involves matching between imaging ambiences like visible light 
photographs in gallery dataset to Probe images that include NIR, thermal, sketches and mugshot. When only a 
particular image ambience is available for querying a large database like visible light photographs called 
Mugshots, the face image captured during night time like infrared imaging cannot be useful to suspect a 
criminal act. This led to the stimulus behind heterogeneous face recognition. Though there is remarkable 
development in Face recognition systems, the Commercial off-The Shelf (COTS) face recognition systems were 
not able to handle HFR Scenarios. 

In this paper we propose a collective way to Heterogeneous Face Recognition [1] which, 

a) Make recognition in the probe and gallery ambiences using multiple feature descriptors, 

b) Attain Accuracy on different HFR Scenarios, 

c) Does not demand any feature descriptor to changes in image prototype 

II. RELATED WORK 

Zhang Wei, Xiaogang Wang, Xiaooua Tang proposed a robust algorithm called 45ultiscale Markov 
Random Field(MRF) to synthesize a face sketch and face photo taken in different lighting condition and 
different pose. They achieved robustness to lighting and pose variations in three steps. First, they introduced 
shape priors to specific facial components to reduce distortions and artifacts created due to variations in pose 
and lighting. Second, to find candidates to the patches of sketch to a given photo, they produced metrics and 
patch descriptors that are vivacious to lighting variations. Third, they used gradient compatibility and intensity 
compatibility which are smoothing term measures to match neighboring sketch patches on the MRF network 
[2]. 

Brenden Klare, Zhifeng Li and Anil K Jain have addressed the problem of matching forensic sketch 
and mugshot images in a gallery. They introduced Local Feature-based Descriminant analysis (LFDA) to solve 
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this problem. Both sketches and photos were represented using 46ultiscale Local Binary Patterns, Histograms 
and Gradient location in LFDA. The dataset contained 10,159 images of forensic sketches in relation to mugshot 
gallery images [3]. 

Brendan F klare and Anil K Jain proposed a novel method of heterogeneous face recognition that uses 
a common feature based representation for both NIR images as well as VIS images. A robust approach to face 
recognition with unconstrained illumination is to match near infrared face images to visible light face images. 
They performed linear discriminant analysis on a collection of random subspaces to learn discriminant 
projections. They matched NIR and VIS images by (i) using sparse representation classification, (ii) directly 
using the random subspace projections [4]. 

Juwei Lu, Konstantinos N Plataniotis, Anastasios N Venetsanopoulos introduced SVM, kernel PCS, 
GDA for pattern regression and classification tasks. The small sample size problem which was a drawback in 
most Face recognition system (FRS) was eliminated by the algorithm called Kernel Direct Local Discriminant 
Analysis (K-D-LDA) [5]. 

Kathryn Bonnen, Brendan Klare, Anil Jain later introduced Component Based Representation for Face 
Recognition to obtain accuracy over occluded face images and enhance robust to various facial poses. They first 
extracted facial landmarks, then they cropped images and obtained feature vector and represented the facial 
components using Local Binary Pattern [6]. 

III. PRELIMINARY PROCESS OF AN IMAGE AND ITS REPRESENTATION 

Feature based representation is the initial representation of a face image. The human visual processing 
system uses local feature descriptors for the proposed representation of the face. 

A. Normalization of an image geometrically: 

Geometric Normalization is the first step to normalize the images of the heterogeneous face using feature 
descriptors in regard to the location of the eyes. In this step, rotation, scale effectiveness, variations in 
translations are reduced [1]. 

We normalized face image geometrically by a) rotating the set of angle between eyes by planar rotation to 
0 degrees, b) ascending the distance between the two pupils to 75 pixels of the image, c) cropping the images 
with eyes placed horizontally centered and vertically placed at row 1 15 of total height of 250 pixels and width of 
200 pixels. 

B. Filtering of Image: 

We use three different filters to filter the face images. These filters reduce the appearance variations 
between image realm and intensity variations within an image domain thus enable denoising in the image. The 
filters are elucidated below: 

1) Difference of Gaussian (DoG): 

To improve the performance of face recognition for varying illumination we use DoG which is a feature 
enhancement algorithm. It subtracts blurred image from original less blurred image. It is commonly used in 
detecting of blob in SIFT. Its main job is to sharp the edges of an image. 

Let us consider, 'a' to be Gaussian filter image of width ol and 'b' to be Gaussian filter image of width o2 
(o2>ol) 

Then DoG = b-a. Here we have taken a 1=2 and o2=4. 

2) Center Surround Divisive Normalization ( CSDN): 

The value of each pixel is divided by the mean pixel value. The mean pixel value is taken from s x s 
neighborhood surrounding the pixel. Here we take s=16. 

3) Gaussian: 

This is the smoothing filter that removes noise from a high spatial frequency. The width used here was 

a 2. 

C. Local Descriptor Representation: 

After normalization and filtration, we extract local feature descriptors across the face from uniformly 
distributed patches. SIFT feature descriptor is widely used in face recognition for effective matching of sketch to 
VIS and NIR to VIS. LBP features have a successful history in face recognition which is applied to several HFR 
matching scenarios[l]. 

SIFT and LBP features can describe the face images and its structures even if there is a minor external 
variation. An image patch is described in each feature descriptor as a d dimensional vector which is normalized 
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to a sum of one. The face image is divided into a size of 32 x 32 set of N overlapping patches. Each patch 
overlaps its neighbors both horizontally and vertically by 16 pixels. So a total of 154 total patches are got from a 
face image of size 200 x 250. Multiscale local binary patterns are used in place of LBP which is a variant of 
LBP descriptor. MLBP over here has radii r= { 1, 3, 5, 7 }. 

Let us consider, I to be the normalized and filtered face image, f FD (I,a) as local feature descriptor 
extracted from image I at patch a, 1 < a <N using image filter F and feature descriptor D, F d as DoG image 
filter, F c as CSDN image filter, F a as Gaussian image filter, D m as MLBP descriptors, D s as SIFT descriptors. 

Then we arrive to the result by using, 

f F .D(I)=[fF.D(I,l) T ,...,fF.D(I,N) T ] T , (1) 

which is a combination of all N feature descriptors. Therefore by using three filters and two descriptors, 
we have six representations for face image I, f Fd ,D m (I), f Fc ,D m (I), f Fg ,D m (I), f Fd ,D s (I), f Fc ,D s (I), f Fg ,D s (I). 

IV. ARCHITECTURAL VIEW OF PROPOSED SYSTEM 

In this Framework, We obtain a Visible Light Photograph and Thermal Photograph from the Database. 
There is separate database for Visible light photographs and Thermal photographs which has image of one 
person taken in different environment. Those images contain noise which has to be preprocessed. Preprocessing 
is done using Difference of Gaussian (DoG) filter, CSDN, Gaussian filter for better denoised image. The 
Denoised image uses SIFT (Scale Invariant Feature Transformation) for extracting local Features. SIFT extracts 
interesting points from the image which provides feature description [1]. From training image SIFT description 
is extracted to identify the object while trying to locate the object in a test image containing many other objects. 
For a dependable recognition feature has to be extracted from the training image which has to be detectable even 
under changes in illuminations, noise, and image scale. Those interesting points usually lie as object edges on 
high-contrast regions of the image. 

We define Local Binary Pattern for each image for differentiating uniform and non -uniform patterns in an 
image. The idea behind is to calculate LBP-code for every pixel of an image [3]. The incidence of each possible 
pattern in the image is conserved. The label which is also known as histogram of these patterns forms a feature 
vector which is a representation for the texture of the image. To measure the similarity between the images, the 
distance between the histograms is calculated. 

After LBP we use Gabor Filter and SVM for feature extraction and Robustness. Gabor filter is used for 
edge detection which is a linear filter. Frequency and orientation representations of Gabor filters are much as 
same to those of the recognition of eye of a human, and they have been found to be particularly appropriate for 
texture representation and discrimination. SVMs can efficiently perform a non-linear classification which maps 
the inputs in a high dimensional feature spaces called to be kernel trick. The images matched are from probe and 
gallery dataset and the results are analysed. 
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Fig 4.1 Architectural View of Proposed System 
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V. ALGORITHM USED 

A. Gabor Filter: 

There is a set of Gabor filters with different frequencies and orientations which is used for feature 
extraction. A 2D Gabor filter is a Gaussian kernel function that is modulated using sinusoidal plane wave. 

The impulse response is defined using sinusoidal wave multiplied by a Gaussian function. Due to 
multiplication convolution property according to Convolution theorem, the Fourier transform of Gabor filter's 
impulse response is the convolution of the Fourier transform of the harmonic function and Gaussian function. 
Gabor filter has both real and imaginary component which represents orthogonal directions. 

Complex form of Component 

g(x,y;A,e,\|/,a,y)=exp(- X ) exp (i {in j + if ) ) 

Real 

g(x,y;X,0,v|/,a,y)= exp(- * ) cos {in T + v) 

Imaginary 

g(x,y;l,0,v|/,a,y)= exp(- * ) sin {in r + if) 

where 

x'= x cos 0 + y sin G 
y'= - x sin 0+ y cos 0 

here, 

X - wavelength of the sinusoidal factor, 

0 — orientation of the normal to the parallel stripes of a Gabor 

function, 
\|/ - phase offset, 

o - sigma/standard deviation of the Gaussian envelope, 
y - spatial aspect ratio 

B. Support Vector Machine (SVM): 

Support Vector Machines are used in Machine Learning which is associated with supervised learning that 
analyse data and recognize patterns for classification and regression analyses. SVM can perform non linear 
classification very efficiently by mapping inputs into high dimensional feature space called Kernel Trick. 

Every linear dot product is replaced with non linear kernel function for non linear classification to fit the 
maximum margin hyper plane in the transformed featured space. Common Kernels used in SVM are as follows: 

a) Polynomial (homogeneous): k (x ; , Xj) = (x ; . Xj) d 

b) Polynomial (inhomogeneous): k (x ; , Xj) = (Xj . Xj+l) d 

c) Gaussian radial basis function: 

k (x ; , Xj) = exp(-y || x ; - Xj II 2 ), for y > 0, parametrized to y = 1/2 o 2 

VI. SIMULATION AND RESULTS 

A. Input Image: 

We take two images as input for matching. One image is RGB image or visual image from Gallery dataset 
and the other one is Thermal Image from probe dataset. 




Fig 5.1 Input Image RGB 
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Fig 5.2 Input Image Thermal 

B. Preprocessing of Image: 

Preprocessing is the process of removing unwanted noise from the image. We use CSDN, Gaussian and 
DoG along with SIFT to remove noise and enhance the image. SIFT can handle images in various pose and 
angles. 




Fig 5.3 Gaussian SIFT image 




Fig 5.4 CSDN SIFT image 




Fig 5.5 Difference of Gaussian (DoG) Image 
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C. LBP of the Image: 

We use Local Binary Pattern algorithm (LBP) along with CSDN, Gaussian and DoG to divide facial 
components into small regions from which histograms are extracted and concatenated into a single, spatially 
enhanced feature histogram. 




Fig 5.6 LBP of the Image 

D. Gabor and SVM: 

We propose a novel approach for feature extraction and robustness called G-SVM (Gabor -Support 
Vector Machine). Gabor helps in edge detection where as SVM helps in Classification of images. The image 
which is matched gives us an output of image matched else it gives us an output of image mismatched. 
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Fig 5.6 Matched Images 




Fig 5.7 Mismatched Imagi 
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E. Results: 

The graph shows the retrieval rate for thermal image which is higher in our proposed system than in the 
existing systems. 
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Fig 5.8 Graph of Rank vs Retrieval rate of Thermal IR 
VII. CONCLUSION 

We have presented a hybrid framework for face recognition based on local binary pattern, multiple 
Gabor filter and SVM. Designing a good filter and classifier is a crucial step for any successful face recognition 
system. An average recognition rate of (96.28%) is achieved under environmental variations. This means that 
our approach achieves a high recognition rate compared to other approaches in published literature. Using 
multiple Gabor filters rendered the method robust to face variations because each filter has specific property to 
extract. In addition using generalization property of bagging classifier increased the recognition rate in presence 
of face class variations. SVM has supported in classification and retrieval. We believe that face recognition 
under varying conditions is still an interesting area of research, and we anticipate that there will be many further 
advances in this area. 
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